United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 13-1450 
www.uspto.gov 



I ATTORNEY DOCKET NO. | CONFIRMATION NcT 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



09/496,086 



02/01/2000 



04/26/2005 



21253 7590 

CHARLES G. CALL 

68 HORSE POND ROAD 

WEST YARMOUTH, MA 02673-2516 



Raja Chatterjee 



99,028 



5705 



EXAMINER 



SMITH, PETER J 



ART UNIT 



PAPER NUMBER 



2176 

DATE MAILED: 04/26/2005 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

09/496,086 


Applicant(s) 

CHATTERJEE ET AL. 


Pvominar 

Peter J Smith 


Art Unit 

2176 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address ~ 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )E] Responsive to communication(s) filed on 31 January 2005 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-17 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) S Claim(s) 1-17 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1 ) □ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5 ) D Notice of Informal Patent Application (PTO-152) 

Paper No(s)/Mail Date . 6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 20050421 



Application/Control Number: 09/496,086 Page 2 

Art Unit: 2176 

DETAILED ACTION 

1. This action is responsive to communications: amendment filed 1/31/2005. 

r 

2. Claims 1-17 are pending in the case. Claims 1, 8, and 10 are independent claims. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 1-2, 4-8, and 10-17 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Nelson et al. (hereinafter "Nelson"), US 6,243,713 Bl filed 8/24/1998. 

Regarding independent claim 1, Nelson teaches analyzing a multimedia document to 
identify at least one markup tag containing a reference to a given one of resources in fig. 1-2, fig. 
4, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 9 line 7. Nelson shows in the col. 
5 lines 39-51 that the multimedia document can be an HTML document and HTML documents 
generally represent Web pages, thus the scope of Nelson includes multimedia Web page 
documents. Nelson also specifically indicates in col. 5 lines 31-38 and col. 8 lines 26-27 that the 
multimedia document may be a downloaded Web page. Nelson teaches selecting and executing 
a media processing program for analyzing the content of the multimedia data supplied by the 
given one of the resources to generate metadata describing the content in fig. 2, fig. 4-7, and col. 
9 line 9 - col. 14 line 55. The cited text section described how Nelson generates metadata for 
each of the types of multimedia content components. Nelson teaches formatting the metadata 
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into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 

Nelson does not teach combining the Web page and the generated annotation (Nelson 
calls it a token) to form an enhanced Web page. As stated earlier, Nelson does teach that all of 
the tokens extracted from a particular document are linked to that document through use of a 
unique document ID in col. 15 lines 24-36. Therefore, tokens of text, image, video, audio, and 
other data are all linked together to represent a single document through use of the unique 
document ID. Thus, this is essentially the same in function, but only similar in form to the 
claimed limitation. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have modified Nelson to have created the claimed invention. It would 
have been obvious and desirable to have modified Nelson to have inserted the token 
representations of image, video, audio, and other multimedia components into the text of the 
multimedia document so that the text did not have to be processed, thus allowing the extraction 
software design to be simpler. 

Regarding dependent claim 2, Nelson teaches determining the particular data type of 
the multimedia data supplied by the given resource and selecting a processing program for 
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analyzing multimedia data formatted in accordance with the particular data type in fig. 2, fig. 4, 
and col. 3 lines 21-27. 

Regarding dependent claim 4, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson teaches that the position data may be used in creating the tokens, thus 
Nelson teaches that the additional metadata may be supplied by a source other than the content of 
the multimedia data. 

Regarding dependent claim 5, Nelson teaches wherein at least some of the additional 
metadata includes information obtained from the one markup tag in fig. 2, fig. 4-7, and col. 9 line 
9 - col. 14 line 55. Nelson indicates in col. 9 line 9 - col. 10 line 37 that the tags may be used to 
identify a position and a name for the identified multimedia component. 

Regarding dependent claim 6, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson does not specifically teach wherein the given resource is accessed 
through the operating system of the computer which provides the given resource and wherein at 
least some of the additional metadata includes information obtained from the operating system. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have modified Nelson to have created the claimed invention. Nelson teaches the 
multimedia content components may be referenced from the multimedia document and thus a 
valid reference would be a multimedia content component from the operating system. It would 
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have be obvious and desirable to have obtained a referenced resource from the operating system 
and used metadata from the operating system to describe the resource so that multimedia 
components could have taken advantage of the local computer's operating system resources. 

Regarding dependent claim 7, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson teaches in col. 5 lines 39-51 that the multimedia component maybe 
referenced by tags in the multimedia document. If the reference indicates a resource located on 
the Internet, Nelson is then going to obtain additional information about the resource via the 
Internet in forming the characterization token. 

Regarding independent claim 8, Nelson teaches a parser for identifying markup tags in 
a Web page in fig. 1-2, fig. 4, col. 2 lines 19-37, col. 5 lines 1 1-51, and col. 8 line 1 - col. 9 line 
7. Nelson shows in the col. 5 lines 39-51 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches analyzing the content of multimedia data identified by markup tags to generate metadata 
describing the multimedia data in fig. 2, fig. 4-7, and col. 9 line 9 - col. 14 line 55. The cited 
text section described how Nelson generates metadata for each of the types of multimedia 
content components. 

Nelson teaches translating the metadata into a character-based text annotation describing 
the multimedia data in fig. 6-8 and col. 3 lines 19-59. Nelson teaches formatting the metadata 
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into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 

Nelson does not teach storing the combination of a copy of the Web page and the 
annotation to form an enhanced Web page suitable for processing by text-based indexing and 
searching facilities. As stated earlier, Nelson does teach that all of the tokens extracted from a 
particular document are linked to that document through use of a unique document ID in col. 15 
lines 24-36. Therefore, tokens of text, image, video, audio, and other data are all linked together 
to represent a single document through use of the unique document ID. Thus, this is essentially 
the same in function, but only similar in form to the claimed limitation. It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to have modified 
Nelson to have created the claimed invention. It would have been obvious and desirable to have 
modified Nelson to have inserted the token representations of image, video, audio, and other 
multimedia components into the text of the multimedia document so that the text did not have to 
be processed, thus allowing the extraction software design to be simpler. 

Regarding independent claim 10, Nelson teaches identifying one or more markup tags 
in a Web page which respectively identify one or more external resources which provide 
multimedia data in fig. 1-2, fig. 4, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 9 
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line 7. Nelson shows in the col. 5 lines 39-51 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches generating metadata which describes the multimedia data in fig. 2, fig. A-l\ and col. 9 
line 9 - col. 14 line 55. The cited text section described how Nelson generates metadata for each 
of the types of multimedia content components. 

Nelson teaches translating the metadata into a character-based text annotation describing 
the multimedia data in fig. 6-8 and col. 3 lines 19-59. Nelson teaches formatting the metadata 
into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 

Nelson does not teach inserting the annotation into the Web page to form an enhanced 
Web page suitable for processing by a character-based text processing system. As stated earlier, 
Nelson does teach that all of the tokens extracted from a particular document are linked to that 
document through use of a unique document ID in col. 15 lines 24-36. Therefore, tokens of text, 
image, video, audio, and other data are all linked together to represent a single document through 
use of the unique document ID. Thus, this is essentially the same in function, but only similar in 
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form to the claimed limitation. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to have modified Nelson to have created the claimed invention. 
It would have been obvious and desirable to have modified Nelson to have inserted the token 
representations of image, video, audio, and other multimedia components into the text of the 
multimedia document so that the text did not have to be processed, thus allowing the extraction 
software design to be simpler. 

Regarding dependent claim 11, Nelson teaches first identifying markup tags in a Web 
page and extracting the uniform resource locator (URL) of one of the external resources from the 
at least selected ones of the markup tags in fig. 1-2, fig. 4-7, col. 2 lines 19-37, col. 5 lines 11-51, 
and col. 8 line 1 - col. 14 line 55. Nelson shows in the col. 5 lines 39-51 that the multimedia 
document can be an HTML document and HTML documents generally represent Web pages, 
thus the scope of Nelson includes multimedia Web page documents. Nelson also specifically 
indicates in col. 5 lines 31-38 and col. 8 lines 26-27 that the multimedia document may be a 
downloaded Web page. Nelson teaches in col. 5 lines 39-51 that the multimedia component may 
be referenced by tags in the multimedia document. The reference to a multimedia component in 
an HTML Web page is a URL. 

Regarding dependent claim 12, Nelson teaches retrieving multimedia data from one or 
more external resources and analyzing the content of the multimedia data to extract the metadata 
therefrom in fig. 1-2, fig. 4-7, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 14 line 
55. Nelson shows in the col. 5 lines 39-5 1 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
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38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches in col. 5 lines 39-51 that the multimedia component may be referenced by tags in the 
multimedia document. The reference is to an external resource from which the multimedia data 
is retrieved and the metadata extract therefrom. 

Regarding dependent claim 13, Nelson teaches identifying the data type of the 
multimedia data from each of the resources and then selecting a processing routine for 
multimedia of the identified data type from each of the resources in fig. 2, fig. 4, and col. 3 lines 
21-27. 

Regarding dependent claim 14, Nelson teaches indexing the multimedia document into 
a unified multimedia index to equally represent in a normalized fashion all of the tokens of the 
multimedia document and provide access to the Web page in response to queries expressing one 
or more attributes expressed in the text annotation in fig. 8-9 and col. 14 line 56 - col. 15 line 58. 
Each of the tokens is linked to the document from which it was extracted through use of a unique 
document ID referencing that document. 

Regarding dependent claim 15, Nelson teaches searching the content of the Web page 
in response to a search request to determine if attributes expressed in the search request are 
contained in the text annotation in fig. 11-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - 
col. 27 line 3. 

Regarding dependent claim 16, Nelson teaches indexing the Web page to provide 
access to the Web page in response to queries expressing one or more attributes expressed in the 
text annotation in fig. 11-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - col. 27 line 3. 
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Regarding dependent claim 17, Nelson teaches searching the content of the Web page 
in response to a search request to determine if attributes expressed in the search request are 
contained in the text annotation in fig. 11-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - 
col. 27 line 3. 

5. Claims 3 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Nelson et al. (hereinafter "Nelson"), US 6,243,713 Bl filed 8/24/1998 in view of Mohan et al. 
(hereinafter "Mohan"), US 6,748,382 Bl filed 12/1/1999 with priority to provisional filed 
1/28/1999. 

Regarding dependent claim 3, Nelson does not teach generating a text data annotation 
expressed in accordance with the Extensible Markup Language. Mohan does teach generating a 
text data annotation expressed in accordance with the Extensible Markup Language in fig. 3 and 
col. 2 lines 16-28. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Mohan into Nelson to have created the claimed invention. 
It would have been obvious and desirable to have enhanced the tokens of Nelson with the XML 
based media asset management descriptions of Mohan to have ensured maximum 
interoperability as described by Mohan in col. 2 lines 27-28. 

Regarding dependent claim 9, Nelson does not teach generating a text data annotation 
expressed in accordance with the Extensible Markup Language. Mohan does teach generating a 
text data annotation expressed in accordance with the Extensible Markup Language in fig. 3 and 
col. 2 lines 16-28. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Mohan into Nelson to have created the claimed invention. 
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It would have been obvious and desirable to have enhanced the tokens of Nelson with the XML 
based media asset management descriptions of Mohan to have ensured maximum 
interoperability as described by Mohan in col. 2 lines 27-28. 

Response to Arguments 
6. Applicant's arguments filed 1/31/2005 have been fully considered but they are not 
persuasive. Regarding Applicant's arguments in pages 6 and 7 that it would not have been 
obvious to have modified Nelson to have created the claimed invention, the Examiner 
respectfully disagrees. Nelson parses the media content and generates tokens describing the 
media content and associates the tokens with the parent document as is described in fig. 8-9 and 
col. 14 line 56 - col. 15 line 58. The difference is that the tokens are maintained separately and 
have an associative link to the document from which they were created. The documents which 
Nelson parses and indexes can be HTML web documents. The HTML specification provides for 
meta-tag annotation of the document to provide existing popular Web search facilities with a 
way to automatically index such documents. Since Nelson teaches how to extract character- 
based annotation data from the multimedia components of the web page, the Examiner believes 
it would have been obvious to have recombined the extracted character data with the Web page 
using the HTML annotation meta-tags capability. Thus, the Examiner maintains the rejection of 
the claimed invention as being obvious over the teachings of Nelson. 

Regarding Applicant's arguments in pages 7 and 8 that Nelson does not teach acquiring 
additional data to describe the media data from a source other than the content of the media data 
itself, the Examiner respectfully disagrees. Nelson teaches that the position data may be used in 
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creating the tokens, thus Nelson teaches that the additional metadata may be supplied by a source 
other than the content of the multimedia data. Furthermore in regard to claim 6, the Examiner 
believes it would have been advantageous to have acquired data through the operating system for 
components dependent on particular hardware configurations and thus believes Nelson would 
have been modified to specifically acquire such metadata. Thus, the Examiner maintains that 
Nelson provides sufficient teachings to enable one of ordinary skill in the art at the time of the 
invention to have created the invention as described in claims 4-7. 

Conclusion 

7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 
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8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 571-272-4101. The 
examiner can normally be reached on Mondays-Fridays 7:00am-3 :30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on 571-272-4090. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



PJS 

4/21/2005 




